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ABSTRACT: This paper describes and evaluates intelligent robotic wheelchair mainly for 
quadriplegic patients. Each year huge number of people suffers with a spinal cord injury and 
approximately half of these injuries result in quadriplegia. The loss of personal mobility is one of the 
major life changes brought on by quadriplegia, mobility becomes a lifelong struggle. Quadriplegics rely 
on power wheelchairs for mobility, but the hands-free controller systems currently available are 
obtrusive and expensive. The objective of this project was to design a power wheelchair with a novel 
control system for quadriplegics with Face Orientations and eye blink. USB camera and eye blink sensor 
was fixed in front of user’s face. Face area was detected based on AdaBoost learning algorithm. Then 
facial landmarks were detected using Elandmark Detector. Finally, face orientations were classified by 
the normalized distance difference in horizontal and vertical axes between eyes, nose and mouth. Face 
orientation, which is used for commands electric wheelchair, consists of frontal, right, left, up and down, 
the eye blink count used to start and stop function. In addition, we can give more independence to the 
disabled person by using this to communicate with the devices in a room for example a fan. 

Keywords: electric wheelchair control, eye blink sensor, face detection, face orientation recognition, 
Elandmark detection, quadriplegic. 



I. Introduction 



Wheelchairs were designed with the idea to help handicapped people to move around and accomplish 
daily life tasks, but what use would it be if the person is severely handicapped? Like quadriplegic patients. 
Many smart wheelchairs have been designed and with more and more advances in technologies newer and 
smarter wheelchairs are coming into the market to help the severely handicapped persons. Our wheelchair is 
also an attempt to make use of simple and easy movements of the eye and face to start, stop and move the 
wheelchair to different directions. In general, most of electric wheelchairs use a joystick for control the 
direction. However there are many researches and developments about electric wheelchair control for support 
the elderly and disabled people who are not able to operate the joystick such as; Ref. [1] proposed a brain 
controlled wheelchair which based on P300 (brain signal). Ref. [2] presented EMG -based hands-free 
wheelchair control with EOG attention shift detection. That bio-signal needs to put bio-sensor on user body. So 
it is not being comfortable for user. Thus, Non-contact electric wheelchair control based on face orientation 
from camera is developed. Ref. [3] proposed head gesture based control of a wheelchair using Boosted Cascade 
of Simple Features integrated with Camshift object tracking algorithm to achieve face region. After that face 
orientation was estimated by using nose template matching. Ref. [4] presented face and mouth shape 
recognition for wheelchair control. Face region was detected based on AdaBoost learning algorithm. The eye 
regions are localized by Neural Network based texture classifier. And then the mouth is localized by edge 
information. Ref. [5], [6], the changes in the darkness area of the both nostrils was utilized for recognition of 
the face orientations. Upward and downward face determine by increasing and decreasing of darkness areas of 
both nostrils. Difference between two nostril’s areas is used for detecting left and right orientation. Ref. [7] 
developed head gesture recognition for wheelchair control by comparing the location of the lip with fixed 
rectangular windows. AdaBoost learning algorithm is used for face and lip detection. 

In this paper, face orientation recognition is proposed for changing the direction of the 
wheelchair and eye blink count used for both control and changing the direction. The face image was taken to 
laptop through USB camera, which is fixed in front of user and eye blink sensor attached to a spectacle which 
user want to wear. Face area was detected based on Haar-like features and AdaBoost learning algorithm [8]. 
Then the detection of facial landmarks such as both-canthus, nose and mouth were implemented by using 
Flandmark Detector []. After that, the normalized distance difference in horizontal and vertical axes between 
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eyes, nose and mouth were used for classify face orientation to 4 classes consist of frontal, right, left, up and 
down. According to the eye blink count corresponding command will generate. 

II. Face Detection And Facial Landmarks Detection 

1. Face Detection 

Pual Viola and Michael Jones proposed face detection based on AdaBoost learning algorithm, which 
rapidly and achieving high detection rates [8]. This research used Haar-like features as shown in Fig. 1. Two- 
rectangles are shown in Fig. 1(a) and Fig. 1(b), three-rectangle is shown in Fig. 1(c) and four-rectangle is 
shown in Fig. 1(d). 
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Fig. 1. Examples of Haar-like features 



Haar-like features is determined by sum of the pixel in white rectangles and subtracted from the sum 
of pixels in the black rectangles. Each rectangle is extracted from sub windows of sample image. A large 
quantity of features comes out after perform Haar-like features calculation. However, only a few features is 
useful. AdaBoost learning algorithm was used to select a few proper features which are able to separate face 
and non-face sub windows. Finally, cascading of classifier was constructed for increasing detection 
performance while reducing computation time. The cascade classifier at each stage is trained to classify 
training sub windows that passed all previous states. If any sub windows fails in any classifier stage, then it is 
immediately classified as non-face. The sub windows that pass through all cascaded stages are classified as 
face. The schematic of cascade classifier is shown in Fig. 2. 




Fig. 2. Schematic of cascaded classifier 



2. Facial Landmarks Detection 

After face area was detected by previous section. Flandmark Detector [] is the method that estimated 
set of facial landmarks such as both-canthus (s5, si, s2 and s6), nose (s7) and mouth-corner (s3, s4). The 
landmarks configuration is identified by scoring function. The scoring function is defined as sum of Local 
Appearance model and Deformation cost. In Local Appearance model term, Local Binary Pattern (LBP) 
pyramid [10] was used as a feature that characterizes the image texture. In Deformation cost term, the 
quadratic function of a displacement vector between landmarks position were used to representation [10]. That 
describes the distance and direction depends on relative positions as shown in Fig. 3(a). For example, s5 (begin 
arrow) with respect to si (end arrow). Each landmark is search in region as shown in Fig. 3(b). Size of each 
region was determined by experiment. 
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(a) 

Fig. 3. (a) Graph constraints (b) Components 



(b) 



Finally, the landmarks position was calculated by maximizing the scoring function. Fig. 4 show block 
diagram of Flandmark Detector. The joint parameter vector was learned by Structure Output Support Vector 
Machine algorithm. 




1 



Landmark positions 

Fig. 4. Flandmark Detector block diagram 



III. Feature Extraction 

After the position of facial landmarks (si to s7) were detected by Flandmark Detector. These positions 
were used to calculate the distance between eyes, nose and mouth in horizontal and vertical axes as shown in 
Fig. 5. However, the distances are normalized by distance of user frontal face because each person has different 
facial characteristic. 

dx dx 

m~ n le-n 




Fig. 5. Distance between eyes, nose and mouth in horizontal and vertical axes 

First, the position of eyes and mouth are identified by center of their corners as in (l)-(3). And the nose 
position is equal to s7 position as shown in (4). 
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Where re, le, m and n are the positions of right eye, left eye, mouth and nose respectively, x ,y denote the 
coordinates on horizontal and vertical axes, respectively. Next, calculate the horizontal distance from right eye 
to nose, dxre-n, and left eye to nose, dxle-n , as shown in (5), (6). 

dxre — n = lore? — xn\ (5) 

dxle — n = bde — xn\ (6] 

The Horizontal distance difference Ax calculated by, (7). 

Ax = (qxtp — n) — Cdxte — n) (7) 

Assume that, the horizontal distance from right eye and left eye to nose are equal, while user’s face turns 
to frontal. So the frontal horizontal distance is calculated by sum of the horizontal distance and divides by two 
as (8). 

I xlef — xnf I ■+■ I xr&f — xn/I 

dxe-nj = m 

And Axnorm is the normalizd horizontal distance difference which is calculated by (). 

Ax — dxe — n, / 

Axno'rm. = — (9) 

dxe — n,f J 

Assume that, right and left eyes are in the same plane. Equation (10) is used to find the vertical distance 
from eye to nose. 

dyte — n = lyte — ynl (10) 



And the vertical distance from mouth to nose is calculated by (1 1). 

ezym — n = \ym — yn\ (11) 

The vertical distance from right eye and left eye to nose of frontal face are equal. So the frontal vertical 
distance from eye to nose is define as (12). 

dyle — -n, f = [ylef — ynf \ (12) 

And the frontal vertical distance from mouth to nose is calculated by (13). 
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dym —n f f= lym/ — ynf \ 



(13) 



Then the normalize! vertical distance difference, Aymorm, is determined by (14). 



fdyle — n — dyls — n, f\ /dym — n — dym — f \ 



( dyle — n — dyle — n, f\ f\ 
dyl * -n.f M 



dym —n f f 



(14) 



From the hypothesis, changes of normalized horizontal distance difference, Axnorm is able to 
separate right-turnand left-turn from frontal face. When face is turned to right, Axnorm will be decreased. 
And when face is turned to left, Axnorm will be increased. Correspondingly, changes of normalized vertical 
distance difference is ableto separate up-turn and down-turn from frontal face by Aynarm . 



IV. Eye Blink Sensor 

The wheel chair can be started and stopped by the eye blink movements. Eye blink sensor senses 
whether eye is open or closed. The eye-blink sensor works by illuminating the eye and/or eyelid area with 
infrared light and then monitoring the changes in the reflected light using a phototransistor and differentiator 
circuit. It has same principle as in all Infra-Red proximity sensors. The basic idea is to send infra-red light 
through IR-LEDs, which is then reflected by any object in front of the sensor. Then pick-up the reflected IR 
light. For detecting the reflected IR light, we are going to use another IR-LED, to detect the IR light that was 
emitted from another LED of the exact same type! This is an electrical property of Light Emitting Diodes 
(LEDs) which is the fact that a LED Produce a voltage difference across its leads when it is subjected to light. 



5V (High) -► LED ON When Eye is close. 

OV (Low) — ► LED OFF when Eye is open. 





Fig. 6. Flowchart Representing the Working of the Eye Blink Sensor 

There are 2 states for the wheelchair State l(Run) and State 2(Stop), if the wheelchair is in State 0 
and the eye is blinked 5 times or more it starts and moves to State 1 . Similarly if the wheelchair is in State 1 
and the eye is blinked 5 times or more it moves to State 0 and stops. 
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V. Implementation 

1. Motors and Motor Driver 

The prototype chair is implemented with a small chair and we use 60rpm motors to move the chair. 
12V rechargeable battery is used to run the motors. The truth table of the motors and the wheels are shown 
below. 



TABLE 1. Truth Table Representing the Working of the Motor 



Input One 


Input Two 


Output 


0 


1 


Forward 


1 


0 


Backward 


1 


1 


Stop 


0 


0 


Stop 



We have connected 2 wheels together and each is controlled simultaneously, the truth table becomes 
as shown in TABLE 2. 

TABLE 2. Truth Table for Controlling Two Wheels Simultaneously 



Motor 1 


Motor 2 


Direction 


Input 1 


Input 2 


Input 1 


Input 2 


0 


1 


0 


1 


Forward 


1 


0 


1 


0 


Reverse 


0 


1 


0 


1 


Right 


1 


0 


0 


1 


Left 



2. Microcontroller 

All logical operations are done in there. The PIC16F877A micro controller is used. The main features 
are; 256 bytes of EEPROM data memory, self-programming, an ICD, 2 Comparators, 8 channels of 10-bit 
Analog-to-Digital (A/D) converter, 2 capture/compare/PWM functions, the synchronous serial port can be 
configured as either 3 -wire Serial Peripheral Interface or the 2- wire Inter-Integrated Circuit bus and a 
Universal Asynchronous Receiver Transmitter (US ART). All of these features make it ideal for more advanced 
level A/D applications in automotive, industrial, appliances and consumer applications. The main thing is, it 
has flash memory so we can rewrite and delete the program also easy to program (only 35 single word 
instructions). It takes 200nanosecond for executing an instruction. According to the eye blinks and face 
orientation it generates commands. And also controls the movement of wheels by giving instructions to motor 
driver. 
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EYE BLINK 
SENSOR 



Fig. 7.Genral Block Diagram 

VI. Conclusion 

In this paper, eye blink and face orientation controlled wheelchair is proposed mainly for quadriplegic 
patients. USB camera was used as the input device to laptop and eye blink sensor attached to a spectacles 
which is directly given to the Microcontroller. The system was begun with a start and stop command which is 
given by eye blink counts. The direction is determined by face detection based on Haar-like features and 
AdaBoost learning algorithm. Then Flandmark Detector was used for facial landmarks detection composed of 
both-canthus, nose and mouth corners. The facial landmarks coordinate (si to s7) was used to locate left and 
right eyes, nose and mouth position. After that, the distance between eyes, nose and mouth were calculated. 
The frontal horizontal and vertical distance was provided for normalization. Finally, the normalized distance 
difference in horizontal and vertical axes is the features for face orientation classification. My future work is to 
operate devices in a room using this system and using Relay and RF module. 
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